Functional safety concept for a 3d interior observation camera

ABSTRACT

An imaging system for a land vehicle may include an imaging sensor, an evaluation device, and an output interface. The imaging sensor may be configured to acquire first signals and second signals, wherein the first signals are obtained via beam pulses provided by a first emitter and reflected on an object and the second signals are obtained via beam patterns provided by a second emitter and reflected on the objected. The evaluation device may be configured to obtain a first distance measurement based on the first signals and a second distance measurement based on the second signals. The evaluation device may also be configured to obtain a third distance measurement via a comparison of the first distance measurement and the second distance measurement.

RELATED APPLICATIONS

This application claims the benefit and priority of German PatentApplication DE 10 2018 204 902.5, filed Mar. 29, 2018, which isincorporated by reference herein in its entirety.

TECHNICAL FIELD

The present disclosure relates to an imaging sensor for a land vehiclefor receiving distance information.

BACKGROUND

Interior cameras for land vehicles are well known from the prior art.Such interior cam-eras provide a two or three-dimensional image and/or atime sequence of such images of the vehicle interior. In particularinterior cameras provide such images and/or sequences of images ofpersons that are arranged in the interior of the land vehicle inparticular of a vehicle driver, a front-seat passenger and/or a furtherpassenger. One well known type of interior cameras are so-called time offlight cameras, which, based on a light propagation between the cameraand an object, calculate a distance of the object to the camera.

WO 2014/195020 A1 discloses an imaging system with a time of flightimaging sensor, wherein the system is designed to receive an image of ascene, wherein the scene has been illuminated with at least twodifferent illumination sources.

In the automotive sector interior cameras must comply with thespecifications of the ISO 26262 standard “Road vehicles—Functionalsafety” in order to be protected from breakdowns.

The embodiments of the present disclosure address the problem ofproviding an interior camera for a land vehicle with improved functionalsecurity.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be described in greater detail on the basis of thefollowing figures. The figures show the following:

FIG. 1 shows an exemplary embodiment of an interior camera according tothe invention in a land vehicle.

FIG. 2 shows an exemplary embodiment of an imaging sensor according tothe invention.

FIG. 3 shows an exemplary embodiment of an imaging system according tothe invention.

FIG. 4 shows a schematic representation the invention according to themethod.

DETAILED DESCRIPTION

The embodiments of the present disclosure address the problem ofproviding an interior camera for a land vehicle with improved functionalsecurity. For example, the problem may be solved by an imaging sensorfor a land vehicle for receiving distance information with the featuresdescribed in this specification.

The imaging sensor according to the invention for a land vehicle forreceiving distance information is designed to detect first and secondsignals. The first signals are reflected beam pulses on objectsilluminated with beam pulses of a first emitter. The second signals arereflected beam patterns on objects illuminated with beam patterns of asecond emitter. The imaging sensor has an evaluation device. Theevaluation device is de-signed to obtain a first distance information ofthese objects depending on the first signals in a first distancemeasurement. Further, the evaluation device is designed to obtain asecond distance information of these objects depending on the secondsignals in a second distance measurement. In addition the evaluationdevice is designed to obtain a third distance information of theseobjects depending on a comparison of the first and the second distanceinformation. Moreover, the imaging sensor has an output interface, whichis designed to provide an image of these objects depending on the thirddistance information. The imaging sensor is designed such that it isfunctionally safeguarded by the first and the second distancemeasurement in the event of a breakdown of one of these distancemeasurements.

The subsequent definitions apply for the entire subject matter of theinvention.

An imaging sensor is a sensor that generates an image from incomingsignals, preferably incident light. An imaging sensor of a digitalcamera generates a two-dimensional image of the recorded scene from theincident light on the sensor.

An emitter is an apparatus that provides beams, for example light,preferably in the form of radiation beams. A radiation beam is a numberof beams, wherein the beams are aligned to one other, preferably runningapproximately parallel to one other. The emitter illuminates objectswith the beams. A projector is an emitter.

A beam pulse is a beam that exists in a finite period of time. This timeperiod can be repeated, preferably periodically. Depending on arespective application, the time period ranges from the order ofmagnitude of nanoseconds to the order of magnitude of attoseconds. Atime coded signal that has a time impulse sequence is a beam pulse.

A beam pattern is a geometric form of illumination that is achieved withthese beams by means of a corresponding emitter, which generates thispattern. Patterns are for example geometric shapes such as lines orcircles, for example. Patterns also means a sequence of such shapes. Thepatterns are achieved by means of space coding of signals. In English,illumination with beam patterns is referred to as structured lighting.

In particular, light beams in the infrared wavelength range function asbeams.

An evaluation device is an apparatus that processes incoming informationand outputs a result of this processing. In particular, an evaluationdevice is an electronic circuit, for example such as a central processorunit or a graphics processor.

The first distance measurement is a propagation time measurement. In thefirst distance measurement the evaluation device acquires the time thebeam pulse requires to reach the object from the first emitter, and thetime required to reach the evaluation device from the object as areflected beam pulse. On the basis of this propagation time the distanceof the object to the evaluation device arises as a result with knowledgeof the propagation speed of the beam pulse.

The second distance measurement is a geometric measurement and ispreferably redundant to the first distance measurement. In the seconddistance measurement the evaluation device acquires a geometric changeof the beam pattern reflected on an object in comparison to the beampattern sent with the second emitter. Depending on the expansion anddistance of the object, patterns of reflected beam patterns appearblurred in comparison to patterns of sent beam patterns. For example,the points of a sent pattern of points appear blurred on a distantcurved surface in comparison to a near plane surface. That means,depending on this geometric change the distance to the object isdetermined.

An interface is a device between at least two function units, at whichan exchange of logical parameters, for example data, or physicalparameters, for example electrical signals, occurs, either onlyunidirectionally or bidirectionally. The exchange can be analog ordigital. The exchange can occur wirelessly or wired.

The image that is provided by the output interface is athree-dimensional image, in which the objects are presentedthree-dimensionally. In the process, the distance of the individualobject points is preferably presented as a function of gray scales.

In a normal, that is means error-free operation, a component of a systemcorrectly executes a security relevant function. Functional securitymeans that, in the event of a failure of the component, the system isreliably controlled, in particular either by transitioning to apreviously defined secure state, or by activation of a furthercomponent, which takes over the function the failed component. Systemswith functional security have a low probability of failure. Particularlyin the automotive sector, different security levels are differentiated,which are distinguished in terms of a recommended or stipulatedprobability of failure. These levels are called ASIL levels, orAutomotive Safety Integrity Levels. ASIL A recommends that fewer than1000 breakdowns occur within 109 operating hours. ASIL B recommends thatfewer than 100 breakdowns occur within 109 operating hours. ASIL Cstipulates that fewer than 100 breakdowns occur within 109 operatinghours. ASIL D stipulates that fewer than 10 breakdowns occur within 109operating hours. For example, ASIL D is required for a componentexecuting a function which, in the event of a malfunction leads tosevere injuries of the user and makes a survival of the user improbable,wherein the malfunction can always occur, that means has a highprobability of occurrence, for example while accelerating, braking orsteering, and is difficult to manage.

If the first distance information corresponds with the second distanceinformation, then the third distance information is equal to the firstis equal to the second distance information. If an error occurs with thefirst distance information, that is, if a result of the first distancemeasurement is zero and a result of the second distance measurement isdifferent from zero, then the third distance information is equal to thesecond distance information. If an error in the second distancemeasurement occurs, that is, if the result of the second distanceinformation is 0 and the result of the first distance information isdifferent from 0, then the third distance information is equal to thefirst distance information. Through this redundancy in the distancemeasurement, the imaging sensor is functionally safeguarded. Moreover, aresult of the first distance measurement can be compared with a resultof the second distance measurement to increase the accuracy of theresult.

In one preferred embodiment of the invention the imaging sensor isreferred to as a time of flight sensor. In the case of a time of flightsensor, every pixel of the sensor collects incident light andsimultaneously measures the propagation time that the light requires togo from a source to the object and from the object back to the pixel.Each of the pixels of the time of flight sensors transforms light to anelectrical current. The pixel works with several switches and in eachcase, with a storage element assigned to a switch. In the simplest case,every pixel has two switches and two storage elements. The switches areactuated with the emission of the beam pulse and opened for the timeperiod of the beam pulse, that is, the pulse length. In the process, thecontrol signals of the respective switch are each deferred by a pulselength. If a reflected beam pulse impinges on the pixel with delay, onlya part of the beam pulse reaches the first storage element, the otherpart is gathered in the second storage element. Depending on distance,this changes the ratio of gathered light in the first storage elementchanges to gathered light in the second storage element. By reading outthe pixels and determining the ratio of the signals in the first and inthe second storage element, the distance of the acquired object thenfollows. The function of a time of flight sensor is disclosed forexample in WO 2014/195020 A1.

In the case of the subject matter of WO 2014/195020 A1, only onedistance measurement is obtained with the light of an illuminationsource. The light of the second illumination source is used there onlyto improve the resolution. The advantage of the imaging sensor accordingto the invention is that with an imaging sensor two distancemeasurements are executed independently from each other and, on thebasis of these distance measurements a resulting distance measurement isalways obtained, even in the event of a failure of a distancemeasurement. That means the imaging sensor is functionally safeguardedagainst the failure of one of the distance measurements.

In one development of the invention the evaluation device is designed toobtain color image data of these objects and, depending on these colorimage data and the third distance information to obtain athree-dimensional color image of these objects, wherein the outputinterface is designed to provide this color image. Thus, in particular atime of flight sensor is provided that supplies a three-dimensionalcolor image with distance information. Moreover two-dimensional imagedata of a common two-dimensional imaging sensor are compared with thetwo-dimensional data of the imaging sensor according to the invention toincrease the functional security of the imaging sensor.

Advantageously, the evaluation device is designed to perform acomparison of the first and/or second signals with the color image data.This comparison improves the functional security.

Preferably, the imaging sensor is an imaging sensor of an interiorcamera of the land vehicle and/or the output interface is a humanmachine interface, preferably in the form of a screen of an infotainmentsystem of the land vehicle. In particular for interior cameras whichcontrol the further security relevant apparatuses in a vehicle,depending on the acquired passengers an imaging sensor according to theinvention is advantageous. Such interior cameras must namely befunctionally safeguarded.

The imaging system for a land vehicle for receiving distance informationaccording to the invention has a first emitter. The first emitter isdesigned to illuminate objects with beam pulses. The imaging system alsohas a second emitter. The second emitter is designed to illuminate theobjects with beam patterns. Further, the imaging system has an imagingsensor. The imaging sensor is designed to detect first and secondsignals. In the process, the first signals are beam pulses reflected onthe objects illuminated with the beam pulses. The second signals arebeam patterns reflected on the objects illuminated with the beampatterns. Moreover, the imaging system has an evaluation device. Theevaluation device is designed to obtain a first distance information ofthe objects depending on the first signals in a first distancemeasurement. Moreover, the evaluation device is designed to obtain asecond distance information of the objects depending on the secondsignals in a second distance measurement. In addition, the evaluationdevice is designed to obtain a third distance information of the objectsdepending on a comparison of the first and second distance measurement.Moreover, the imaging system has an output interface. The outputinterface is designed to provide an image of these objects depending onthe third distance information. The imaging system is designed such thatit is functionally safeguarded by the first and second distancemeasurement in the event of a failure of one of these distancemeasurements.

With the imaging system a complete system for imaging is provided thathas the advantages of the imaging sensor according to the invention.

Preferably, the imaging system has an optical device, which is designedto display beams reflected on these objects on the imaging sensor.

An optical device is an optical system that gathers reflected beams on asurface of an imaging sensor of the imaging system.

Preferably, the optical device has an optical band-pass filter whichonly allows those wavelengths to pass through with which theillumination also works. Thus, a majority of the disturbing backgroundlight is eliminated.

Within the scope of the invention, disturbing background light is alsoeliminated by means of the first and second signals of the imagingsensor. The disturbing background light appears both in the first signaland in the second signal and can be easily subtracted.

In one preferred embodiment of the invention the imaging sensor and/orthe optical device are functionally safeguarded by the first and seconddistance measurement, and/or preferably by a comparison of the firstand/or of the second signals with color images of these objects.Preferably, a failure rate of the imaging system is less than 100breakdowns per 109 operating hours of the operating system. With this,an ASIL B level is reached for the imaging system.

The color images are obtained with the optical device of the imagingsystem, so that the imaging system has only one optical path, which isfunctionally safeguarded according to the invention. The color imagescan also be obtained with a second optical device.

Preferably the imaging sensor is an imaging sensor according to theinvention.

A use of an imaging system according to the invention with anenvironment detection system, preferably an interior camera, of a landvehicle, is also in accordance with the invention.

An environment detection system is a system that acquires theenvironment of a vehicle and provides the acquisition to a driver of theland vehicle, in particular within the scope of assisting driving tasksor also in the case of an automated driving operation. Environmentdetection systems are for example, external cameras with correspondingmerging of the collected camera data and an actuation of vehicleactuators depending on said collected camera data.

Advantageously, an imaging sensor of the environment detection system isan imaging sensor according to the invention or an imaging system of theenvironment detection system is an imaging system according to theinvention.

Particularly preferably the environment detection system is an interiorcamera.

The method for functional safeguarding of an environment detectionsystem of a land vehicle according to the invention has the followingsteps:

illuminating objects with beam pulses and obtaining beam pulsesreflected on the objects,

illuminating objects with beam patterns and obtaining beam patternsreflected on the objects,

in a first distance measurement, measurement of propagation times of thebeam pulses and obtaining a first distance information these objectsdepending on the propagation times,

in a second distance measurement, measurement of reflection patterns ofthe beam patterns obtaining a second distance information of theseobjects depending on the reflection patterns,

comparison of the first and second distance information and obtaining athird distance information of these objects depending on the comparison,and

providing an image of these objects depending on the third distanceinformation, wherein the environment detection system is functionallysafeguarded by the first and the second distance measurement in theevent of a failure of one of these distance measurements.

Hence, in addition to an apparatus for functional safeguarding,advantageously a corresponding method is also provided.

Preferably, color images of these objects are obtained and these colorimage data are compared with the beams reflected on these objects toincrease the functional security of the environment detection system.

Particularly preferably, an imaging sensor according to the invention oran imaging system according to the invention is used to perform themethod.

A computer program product according to the invention is designed to beloaded into a memory of a computer and comprises software code sections,with which the method according to the invention can be executed whenthe computer program is running in the computer. Preferably, thecomputer program product is loaded into a memory of the imaging sensoror of the imaging system.

In the following figures identical reference numerals denote identicalreference elements. In the corresponding figures, in each case therelevant reference elements are numbered.

FIG. 1 shows a passenger vehicle as a land vehicle 1. In an interior 6of the land vehicle 1 a passenger is arranged on a driver's seat as anobject 2. The passenger is for example the vehicle driver. The passengercan however also be a front-seat passenger. Passengers, e.g. children,can also be on the back seat.

An interior camera 3 is arranged in the interior 6 as an environmentdetection system 50. The interior camera 3 is designed such that itacquires the entire interior 6. Preferably, for this purpose theinterior camera has a wide-angle lens as an optical device 23. Theenvironment detection system 50 can be connected to further advancedriver assistance systems and vehicle actuators. In particular, in theprocess the data gathered by the individual advance driver assistancesystem are merged with each another. The interior camera 3 acquires theobject 2 and an arrangement of a seat belt 5. The interior camera 3 isin particular connected to an airbag controller. For example, if theinterior camera 3 detects that the vehicle driver is seated against thedriving direction, that is, is arranged with his face and upper bodypointing to the back seat against the driving direction, then theinterior camera 3 controls the airbag controller for the vehicle driversuch that an airbag is not triggered. The interior camera 3 onlyswitches the airbag controller for the vehicle driver to an active stateif the interior camera 3 has detected that the vehicle driver isarranged seated in the driving direction.

FIG. 2 shows an imaging sensor 10. A pixel arrangement 16 of the imagingsensor 10 receives first signals 11 and second signals 12 reflected onthe object 2. The object 2 is illuminated with beam pulses 13 of a firstemitter 21. Moreover, the object 2 is illuminated with beam patterns 15of a second emitter 22. A single beam pulse of the beam pulses 13 has aduration of preferably below 50 ns, in particular a duration of 10 ns.The beam pattern 15 is an arrangement of light points. The pixelarrangement 16 is a two-dimensional arrangement of pixels. The imagingsensor 10 has for example a quadratic shape with an edge length of 45μm. In the process, the pixel arrangement 16 preferably has 200×200pixels.

From the signals obtained with the pixel arrangement 16 an evaluationdevice 30 in a first distance measurement obtains a first distanceinformation of the illuminated object 2. The first distance measurementis a measurement of propagation times of the beam pulses 13. In thefirst distance measurement a distance information of these objects isobtained depending on the propagation times.

In a second distance measurement the evaluation device obtains a seconddistance information of the illuminated object 2. The second distancemeasurement H measures reflection patterns of the beam patterns 15 andobtains a second distance information of the illuminated objectdepending on the reflection patterns. In addition, the signals, whichthe evaluation device 30 receives from the pixel arrangement 16, aremerged with signals of a 2D-imaging sensor 4. The 2D-imaging sensor 2 isa high resolution two-dimensional imaging sensor. The evaluation device30 thus merges the signals that are obtained with the pixel arrangement16, with color image data of the 2D-imaging sensor 14 and obtains a highresolution three-dimensional image with color information. Depending onthe third distance information, which the evaluation device 30 obtainsfrom a comparison of the first distance information with the seconddistance information, this image is provided via an output interface 40.

The 2D-imaging sensor 14 preferably has the same optical device 23 asthe imaging system 20, thus is arranged in the optical path of theimaging sensor 10. Alternatively, the 2D-imaging sensor 14 has its ownoptical device 23.

The output interface 40 is a human machine interface in the form of ascreen of an infotainment system of the land vehicle.

FIG. 3 shows an imaging system 20. The imaging system 20 has a firstemitter 21 and a second emitter 22. The first emitter 21 illuminatesobjects with first signals 11. The first signals 11 are beam pulses witha specified pulse length. The second emitter 22 illuminates objects 2with beam patterns by means of a second signal 12.

The imaging system 20 has in addition an optical device 23. The opticaldevice 23 is a system of optical convergent and divergent lenses.

Further, the imaging system has 20 an evaluation device 30 and an outputinterface 40. The imaging sensor 10 is preferably an imaging sensoraccording to the invention 10. The output interface 40 provides an imageof the object 2 depending on the third distance information obtainedwith the evaluation device 30 of the distance information from acomparison of the first distance information with the second distanceinformation.

FIG. 4 shows a method for the functional safeguarding of an environmentdetection system 50. In the process, objects 2 are illuminated with beampulses 13 in a first step V1. Beam pulses 13 reflected on the objects 2are obtained. In a second step V2 the objects 2 are illuminated withbeam patterns 15. Beam patterns 15 reflected on the objects areobtained. Steps V1 and V2 occur successively or in parallel in terms oftime. In a first distance measurement, propagation times of the beampulses are measured and in a step V3 a first distance information of theobject 2 is obtained depending on the propagation times. In a seconddistance measurement, reflection patterns of the beam patterns aremeasured and in a step V4 a second distance information of the object 2is obtained depending on the reflection patterns. In a step V5 the firstand the second distance information items are compared to each otherand, depending on the comparison, a third distance information of theobject 2 is obtained. Depending on the third distance information, animage of the object 2 is provided in step V6. Parallel to the comparisonof the first and second distance information, in a step V7 color imagesof the objects 2 are obtained. These color images are compared with thebeams reflected on the objects 2 in a step V8.

For the functional security of the environment detection system 50 thedata of the 2D-imaging sensor are compared with the two-dimensionalimage information of the imaging sensor 10. The imaging sensor 10 ispreferably a time of flight sensor.

We claim:
 1. An imaging system for a land vehicle, comprising: animaging sensor; an evaluation device; and an output interface, whereinthe imaging sensor is configured to acquire first signals and secondsignals, wherein the first signals are obtained via beam pulses providedby a first emitter and reflected on an object and the second signals areobtained via beam patterns provided by a second emitter and reflected onthe objected, wherein the evaluation device is configured to obtain afirst distance measurement based on the first signals and a seconddistance measurement based on the second signals, wherein the evaluationdevice is configured to obtain a third distance measurement via acomparison of the first distance measurement and the second distancemeasurement, wherein the output interface is configured to form an imageof the object based on the third distance measurement, and wherein theimaging sensor is configured such that it is functionally safeguarded bythe first distance measurement and the second distance measurement inthe event of a failure of at least one of the first distance measurementand the second distance measurement.
 2. The imaging system according toclaim 1, where the imaging sensor is a time of flight method sensor. 3.The imaging system according to claim 1, where the evaluation device isconfigured to obtain color image data of the object and to obtain athree-dimensional color image of the object based on the color imagedata and the third distance measurement, and wherein the outputinterface is configured to display this color image.
 4. The imagingsystem according to claim 3, where the evaluation device is configuredto perform a comparison of the first signals and the second signals withthe color image data.
 5. The imaging system according to any of claim 1,where the imaging sensor includes a camera within an interior of theland vehicle.
 6. The imaging system according to claim 1, wherein theoutput interface includes a human machine interface with a screenlocated in the interior of the land vehicle.
 7. An imaging system for aland vehicle, comprising: a first emitter, which is configured toilluminate an object with beam pulses, a second emitter, which isconfigured to illuminate the object with beam patterns, an imagingsensor, which is configured to acquire first signals based on the beampulses when they are reflected on the object, wherein the imaging sensoris configured to acquire second signals based on the beam patters whenthey are reflected on the object, and second signals; an evaluationdevice, wherein the evaluation device is configured to obtain a firstdistance a first distance measurement based on the first signals and asecond distance measurement based on the second signals, and wherein theevaluation device is configured to obtain a third distance measurementbased on a comparison of the first distance measurement and the seconddistance measurement; and an output interface, wherein the outputinterface is configured to form an image of the object based on thethird distance information, and wherein the imaging system is configuredsuch that it is functionally safeguarded by the first distancemeasurement and the second distance measurement in the event of afailure of one of the first distance measurement and the second distancemeasurement.
 8. The imaging system according to claim 7, where theimaging system includes an optical device which is configured to displaybeams reflected on the object on the imaging sensor.
 9. The imagingsystem according to claim 8, where the imaging sensor is functionallysafeguarded by at least one of the first and the second distancemeasurements.
 10. The imaging system according to claim 8, where theoptical device is functionally safeguarded by at least one of the firstand the second distance measurements.
 11. The imaging system accordingto claim 7, wherein a failure rate of the imaging system is less than100 breakdowns per 109 operating hours of the imaging system.
 12. Theimaging system according to claim 7, wherein at least a portion of theimaging system is located in an interior of the land vehicle.
 13. Theimaging system according to claim 7, where the imaging sensor is a timeof flight method sensor.
 14. The imaging system according to claim 7,where the evaluation device is configured to obtain color image data ofthe object and to obtain a three-dimensional color image of the objectbased on the color image data and the third distance information, andwherein the output interface is configured to display this color image.15. The imaging system according to claim 14, where the evaluationdevice is configured to perform a comparison of the first signals andthe second signals with the color image data.
 16. The imaging systemaccording to claim 7, where the imaging sensor includes a camera withinthe interior of the land vehicle.
 17. The imaging system according toclaim 7, wherein the output interface includes a human machine interfacewith a screen located in the interior of the land vehicle.
 18. A methodfor functional safeguarding an environment detection system of a landvehicle, the method comprising: illuminating an object with beam pulsesand obtaining beam pulses reflected from the object; illuminating theobject with beam patterns and obtaining beam patterns reflected from theobject; measuring the propagation times of the beam pulses to obtain afirst distance measurement; measuring reflection patterns of the beampatterns to obtain a second distance measurement; comparing the firstdistance measurement and the second distance measurement to obtain athird distance measurement based on the comparison; and forming an imageof the object based on the third distance measurement.
 19. The methodaccording to claim 18, further comprising color image data of theobject, wherein the color image data is compared with the beamsreflected on the object to increase the functional security of theenvironment detection system.
 20. The method according to claim 18,wherein an imaging sensor is included for detecting the beam pulses andthe beam patters, the imaging sensor including a camera.